Multiple base recognition modules that complement each other and a combiner are commonly used in pattern recognition for enhanced results. Combiners are usually trained from data to learn how to combine results from various base modules. The data used to train the combiner may be the same training data set used to train base modules, or a tune data set distinct from the training set. The tune set usually may provide better results as the features generated are closer to the generalization case. The effectiveness of a combiner is typically evaluated on a test data set distinct from both the training and the tune data sets.
This scheme of combiner training and evaluation provides better results when a large amount of training data is available, where base modules can be sufficiently trained, there is sufficient tune data to generate training patterns for the combiner, and there is sufficient test data to accurately evaluate the combiner precision.
However, in many practical pattern recognition problems (e.g. handwriting recognition), there may be a deficiency of training data. The aforementioned traditional method, which partitions data into a training set, a tune set, and a test set may not work well, as it may generate insufficiently trained base modules and combiners. The system is also evaluated with insufficient test patterns.